Generation, administration and analysis of user experience testing

ABSTRACT

Systems and methods for generating, administering and analyzing a user experience study are provided. In particular, intents can be generated from a user experience study by applying one or more screener questions to participants and subjecting the screened participants to one or more tasks. Corresponding clickstreams and success data for each participant engaging in the tasks can be recorded. The success and clickstream data can also be aggregated for all the screened participants as aggregated results. Video data including audio for each of the screened participants can also be recorded.

CROSS REFERENCE TO RELATED APPLICATIONS

This continuation-in-part application is a non-provisional and claimsthe benefit of U.S. Provisional Application of the title “Systems andMethods for the Generation, Administration and Analysis of UserExperience Testing”, U.S. provisional application No. 62/799,646, filedin the USPTO on Jan. 31, 2019, by inventor Mestres et al.

This continuation-in-part application also claims the benefit of U.S.application entitled “System and Method for Unmoderated Remote UserTesting and Card Sorting,” U.S. application Ser. No. 13/112,792, filedin the USPTO on May 20, 2011, by inventor Mestres et al., which claimsthe benefit of U.S. Provisional Application of the same title, U.S.application Ser. No. 61/348,431, filed in the USPTO on May 26, 2010, byinventors Mestres et. al.

This continuation-in-part application additionally claims the benefit ofU.S. Application No. entitled “Unmoderated Remote User Testing and CardSorting,” U.S. application Ser. No. 16/163,913, filed in the USPTO onOct. 18, 2018, by inventor Mestres et al.

All of the above-referenced applications are incorporated herein intheir entirety by this reference.

BACKGROUND

The present invention relates to systems and methods for the generationof studies that allow for insight generation for the usability of awebsite. Generally, this type of testing is referred to as “UserExperience” or merely “UX” testing.

The Internet provides new opportunities for business entities to reachcustomers via web sites that promote and describe their products orservices. Often, the appeal of a web site and its ease of use may affecta potential buyer's decision to purchase the product/service.

Especially as user experiences continue to improve and competitiononline becomes increasingly aggressive, the ease of use by a particularretailer's website may have a material impact upon sales performance.Unlike a physical shopping experience, there is minimal hurdles to auser going to a competitor for a similar service or good. Thus, inaddition to traditional motivators (e.g., competitive pricing, returnpolicies, brand reputation, etc.) the ease of a website to navigate isof paramount importance to a successful online presence.

As such, assessing the appeal, user friendliness, and effectiveness of aweb site is of substantial value to marketing managers, web sitedesigners and user experience specialists; however, this information istypically difficult to obtain. Focus groups are sometimes used toachieve this goal but the process is long, expensive and not reliable,in part, due to the size and demographics of the focus group that maynot be representative of the target customer base.

In more recent years advances have been made in the automation andimplementation of mass online surveys for collecting user feedbackinformation. Typically these systems include survey questions, orpotentially a task on a website followed by feedback requests. Whilesuch systems are useful in collecting some information regarding userexperiences, the studies often suffer from biases in responses, andlimited types of feedback collected.

It is therefore apparent that an urgent need exists for advancements inthe generation, implementation and analysis of studies into userexperiences. Such systems and methods allow for improvements in web sitedesign, marketing and brand management.

SUMMARY

To achieve the foregoing and in accordance with the present invention,systems and methods for generating, administering and analyzing a userexperience study. This enables the efficient generation of insightsregarding the user experience so that the experience can be changed toimprove the customer or user experience.

In some embodiments, the system and methods includes the selection ofparticipants as either ones supplied by the user, or ones the systemprovides. The participants the system provides includes screening alarge pool of participants by a set of basic metrics (age, gender andincome) or by advanced query questions have branched answers. Thesescreener questions may be nested to allow for various participant groupsto be generated. After the participants are screened, they may beinvited to join the study.

The study itself may be a card sorting exercise, survey, tree study,click test, basic navigation, or advanced recorded study. A click testgenerate a ‘heat map’ when the participant is shown a static image andprompted to undergo a task or asked a question. The location and speedthe user clicks on the image is used to generate the heat map. Theadvanced recorded study can present the user with a survey, navigationtask, or any other desired activity. The participant can be recorded(audio and/or video) for downstream analysis. For any navigation aspectsof the study the participant's click flow can also be monitored and usedto populate a click-flow branched chart.

Recordings may be processed for additional analysis. For example machinelearning may analyze video for eye movements and/or emotion, forexample. The analysis can also include transcribing the audio,synchronizing the transcription with the video recording, and allowingfor automatic clip generation when portions of the transcription areselected. The transcriptions may also be searched by keyword,tagged/annotated and these annotations are likewise searchable.

All results, whether they be videos, survey results, click flows, heatmaps, etc. may all be filterable across any participant feature or anystudy validation criteria. Validation criteria includes time taken tocomplete a given action, ending up at a particular URL (or class ofURLs), based upon question answers, or any combination thereof. Thesecriteria can be classified as either a successful completion of thestudy, or a failed attempt. Additionally, the validation criteria canalso include a decision by the participant to abandon the study.

Note that the various features of the present invention described abovemay be practiced alone or in combination. These and other features ofthe present invention will be described in more detail below in thedetailed description of the invention and in conjunction with thefollowing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the present invention may be more clearly ascertained,some embodiments will now be described, by way of example, withreference to the accompanying drawings, in which:

FIG. 1A is an example logical diagram of a system for user experiencestudies, in accordance with some embodiment;

FIG. 1B is a second example logical diagram of a system for userexperience studies, in accordance with some embodiment;

FIG. 1C is a third example logical diagram of a system for userexperience studies, in accordance with some embodiment;

FIG. 2 is an example logical diagram of the usability testing system, inaccordance with some embodiment;

FIG. 3A is a flow diagram illustrating an exemplary process ofinterfacing with potential candidates and pre-screening participants forthe usability testing according to an embodiment of the presentinvention;

FIG. 3B is a flow diagram of an exemplary process for collectingusability data of a target web site according to an embodiment of thepresent invention;

FIG. 3C is a flow diagram of an exemplary process for card sortingstudies according to an embodiment of the present invention;

FIG. 4 is a simplified block diagram of a data processing unitconfigured to enable a participant to access a web site and trackparticipant's interaction with the web site according to an embodimentof the present invention;

FIG. 5 is an example logical diagram of a second substantiation of theusability testing system, in accordance with some embodiment;

FIG. 6 is a logical diagram of the study generation module, inaccordance with some embodiment;

FIG. 7 is a logical diagram of the recruitment engine, in accordancewith some embodiment;

FIG. 8 is a logical diagram of the study administrator, in accordancewith some embodiment;

FIG. 9 is a logical diagram of the research module, in accordance withsome embodiment;

FIG. 10 is a flow diagram for an example process of user experiencetesting, in accordance with some embodiment;

FIG. 11 is a flow diagram for the example process of study generation,in accordance with some embodiment;

FIG. 12 is a flow diagram for the example process of studyadministration, in accordance with some embodiment;

FIG. 13 is a flow diagram for the example process of insight generation,in accordance with some embodiment; and

FIGS. 14-24, 25A-25F, 26, 27A-27D, 28A-28B and 29 are examplescreenshots of some embodiments of the user experience testing system.

DETAILED DESCRIPTION

The present invention will now be described in detail with reference toseveral embodiments thereof as illustrated in the accompanying drawings.In the following description, numerous specific details are set forth inorder to provide a thorough understanding of embodiments of the presentinvention. It will be apparent, however, to one skilled in the art, thatembodiments may be practiced without some or all of these specificdetails. In other instances, well known process steps and/or structureshave not been described in detail in order to not unnecessarily obscurethe present invention. The features and advantages of embodiments may bebetter understood with reference to the drawings and discussions thatfollow.

Aspects, features and advantages of exemplary embodiments of the presentinvention will become better understood with regard to the followingdescription in connection with the accompanying drawing(s). It should beapparent to those skilled in the art that the described embodiments ofthe present invention provided herein are illustrative only and notlimiting, having been presented by way of example only. All featuresdisclosed in this description may be replaced by alternative featuresserving the same or similar purpose, unless expressly stated otherwise.Therefore, numerous other embodiments of the modifications thereof arecontemplated as falling within the scope of the present invention asdefined herein and equivalents thereto. Hence, use of absolute and/orsequential terms, such as, for example, “will,” “will not,” “shall,”“shall not,” “must,” “must not,” “first,” “initially,” “next,”“subsequently,” “before,” “after,” “lastly,” and “finally,” are notmeant to limit the scope of the present invention as the embodimentsdisclosed herein are merely exemplary.

The present invention relates to enhancements to traditional userexperience testing and subsequent insight generation. While such systemsand methods may be utilized with any user experience environment,embodiments described in greater detail herein are directed to providinginsights into user experiences in an online/webpage environment. Somedescriptions of the present systems and methods will also focus nearlyexclusively upon the user experience within a retailer's website. Thisis intentional in order to provide a clear use case and brevity to thedisclosure, however it should be noted that the present systems andmethods apply equally well to any situation where a user experience inan online platform is being studied. As such, the focus herein on aretail setting is in no way intended to artificially limit the scope ofthis disclosure.

The following description of some embodiments will be provided inrelation to numerous subsections. The use of subsections, with headings,is intended to provide greater clarity and structure to the presentinvention. In no way are the subsections intended to limit or constrainthe disclosure contained therein. Thus, disclosures in any one sectionare intended to apply to all other sections, as is applicable.

The following systems and methods are for improvements in naturallanguage processing and actions taken in response to such messageexchanges, within conversation systems, and for employment of domainspecific assistant systems that leverage these enhanced natural languageprocessing techniques. The goal of the message conversations is toenable a logical dialog exchange with a recipient, where the recipientis not necessarily aware that they are communicating with an automatedmachine as opposed to a human user. This may be most efficientlyperformed via a written dialog, such as email, text messaging, chat,etc. However, given the advancement in audio and video processing, itmay be entirely possible to have the dialog include audio or videocomponents as well.

In the following it is understood that the term usability refers to ametric scoring value for judging the ease of use of a target web site. Aclient refers to a sponsor who initiates and/or finances the usabilitystudy. The client may be, for example, a marketing manager who seeks totest the usability of a commercial web site for marketing (selling oradvertising) certain products or services. Participants may be aselected group of people who participate in the usability study and maybe screened based on a predetermined set of questions. Remote usabilitytesting or remote usability study refers to testing or study inaccordance with which participants (referred to use their computers,mobile devices or otherwise) access a target web site in order toprovide feedback about the web site's ease of use, connection speed, andthe level of satisfaction the participant experiences in using the website. Unmoderated usability testing refers to communication with testparticipants without a moderator, e.g., a software, hardware, or acombined software/hardware system can automatically gather theparticipants' feedback and records their responses. The system can testa target web site by asking participants to view the web site, performtest tasks, and answer questions associated with the tasks.

To facilitate the discussion, FIG. 1A is a simplified block diagram of auser testing platform 100A according to an embodiment. Platform 100A isadapted to test a target web site 110. Platform 100A is shown asincluding a usability testing system 150 that is in communications withdata processing units 120, 190 and 195. Data processing units 120, 190and 195 may be a personal computer equipped with a monitor, a handhelddevice such as a tablet PC, an electronic notebook, a wearable devicesuch as a cell phone, or a smart phone.

Data processing unit 120 includes a browser 122 that enables a user(e.g., usability test participant) using the data processing unit 120 toaccess target web site 110. Data processing unit 120 includes, in part,an input device such as a keyboard 125 or a mouse 126, and a participantbrowser 122. In one embodiment, data processing unit 120 may insert avirtual tracking code to target web site 110 in real-time while thetarget web site is being downloaded to the data processing unit 120. Thevirtual tracking code may be a proprietary JavaScript code, whereby therun-time data processing unit interprets the code for execution. Thetracking code collects participants' activities on the downloaded webpage such as the number of clicks, key strokes, keywords, scrolls, timeon tasks, and the like over a period of time. Data processing unit 120simulates the operations performed by the tracking code and is incommunication with usability testing system 150 via a communication link135. Communication link 135 may include a local area network, ametropolitan area network, and a wide area network. Such a communicationlink may be established through a physical wire or wirelessly. Forexample, the communication link may be established using an Internetprotocol such as the TCP/IP protocol.

Activities of the participants associated with target web site 110 arecollected and sent to usability testing system 150 via communicationlink 135. In one embodiment, data processing unit 120 may instruct aparticipant to perform predefined tasks on the downloaded web siteduring a usability test session, in which the participant evaluates theweb site based on a series of usability tests. The virtual tracking code(e.g., a proprietary JavaScript) may record the participant's responses(such as the number of mouse clicks) and the time spent in performingthe predefined tasks. The usability testing may also include gatheringperformance data of the target web site such as the ease of use, theconnection speed, the satisfaction of the user experience. Because theweb page is not modified on the original web site, but on the downloadedversion in the participant data processing unit, the usability can betested on any web sites including competitions' web sites.

Data collected by data processing unit 120 may be sent to the usabilitytesting system 150 via communication link 135. In an embodiment,usability testing system 150 is further accessible by a client via aclient browser 170 running on data processing unit 190. Usabilitytesting system 150 is further accessible by user experience researcherbrowser 180 running on data processing unit 195. Client browser 170 isshown as being in communications with usability testing system 150 viacommunication link 175. User experience research browser 180 is shown asbeing in communications with usability testing system 150 viacommunications link 185. A client and/or user experience researcher maydesign one or more sets of questionnaires for screening participants andfor testing the usability of a web site. Usability testing system 150 isdescribed in detail below.

FIG. 1B is a simplified block diagram of a user testing platform 100Baccording to another embodiment of the present invention. Platform 100Bis shown as including a target web site 110 being tested by one or moreparticipants using a standard web browser 122 running on data processingunit 120 equipped with a display. Participants may communicate with ausability test system 150 via a communication link 135. Usability testsystem 150 may communicate with a client browser 170 running on a dataprocessing unit 190. Likewise, usability test system 150 may communicatewith user experience researcher browser running on data processing unit195. Although a data processing unit is illustrated, one of skill in theart will appreciate that data processing unit 120 may include aconfiguration of multiple single-core or multi-core processorsconfigured to process instructions, collect usability test data (e.g.,number of clicks, mouse movements, time spent on each web page,connection speed, and the like), store and transmit the collected datato the usability testing system, and display graphical information to aparticipant via an input/output device (not shown).

FIG. 1C is a simplified block diagram of a user testing platform 100Caccording to yet another embodiment of the present invention. Platform100C is shown as including a target web site 130 being tested by one ormore participants using a standard web browser 122 running on dataprocessing unit 120 having a display. The target web site 130 is shownas including a tracking program code configured to track actions andresponses of participants and send the tracked actions/responses back tothe participant's data processing unit 120 through a communication link115. Communication link 115 may be computer network, a virtual privatenetwork, a local area network, a metropolitan area network, a wide areanetwork, and the like. In one embodiment, the tracking program is aJavaScript configured to run tasks related to usability testing andsending the test/study results back to participant's data processingunit for display. Such embodiments advantageously enable clients usingclient browser 170 as well as user experience researchers using userexperience research browser 180 to design mockups or prototypes forusability testing of variety of web site layouts. Data processing unit120 may collect data associated with the usability of the target website and send the collected data to the usability testing system 150 viaa communication link 135.

In one exemplary embodiment, the testing of the target web site (page)may provide data such as ease of access through the Internet, itsattractiveness, ease of navigation, the speed with which it enables auser to complete a transaction, and the like. In another exemplaryembodiment, the testing of the target web site provides data such asduration of usage, the number of keystrokes, the user's profile, and thelike. It is understood that testing of a web site in accordance withembodiments of the present invention can provide other data andusability metrics. Information collected by the participant's dataprocessing unit is uploaded to usability testing system 150 viacommunication link 135 for storage and analysis.

FIG. 2 is a simplified block diagram of an exemplary embodiment platform200 according to one embodiment of the present invention. Platform 200is shown as including, in part, a usability testing system 150 being incommunications with a data processing unit 125 via communications links135 and 135′. Data processing unit 125 includes, in part, a participantbrowser 120 that enables a participant to access a target web site 110.Data processing unit 125 may be a personal computer, a handheld device,such as a cell phone, a smart phone or a tablet PC, or an electronicnotebook. Data processing unit 125 may receive instructions and programcodes from usability testing system 150 and display predefined tasks toparticipants 120. The instructions and program codes may include aweb-based application that instructs participant browser 122 to accessthe target web site 110. In one embodiment, a tracking code is insertedto the target web site 110 that is being downloaded to data processingunit 125. The tracking code may be a JavaScript code that collectsparticipants' activities on the downloaded target web site such as thenumber of clicks, key strokes, movements of the mouse, keywords,scrolls, time on tasks and the like performed over a period of time.

Data processing unit 125 may send the collected data to usabilitytesting system 150 via communication link 135′ which may be a local areanetwork, a metropolitan area network, a wide area network, and the likeand enable usability testing system 150 to establish communication withdata processing unit 125 through a physical wire or wirelessly using apacket data protocol such as the TCP/IP protocol or a proprietarycommunication protocol.

Usability testing system 150 includes a virtual moderator softwaremodule running on a virtual moderator server 230 that conductsinteractive usability testing with a usability test participant via dataprocessing unit 125 and a research module running on a research server210 that may be connected to a user research experience data processingunit 195. User experience researcher 181 may create tasks relevant tothe usability study of a target web site and provide the created tasksto the research server 210 via a communication link 185. One of thetasks may be a set of questions designed to classify participants intodifferent categories or to prescreen participants. Another task may be,for example, a set of questions to rate the usability of a target website based on certain metrics such as ease of navigating the web site,connection speed, layout of the web page, ease of finding the products(e.g., the organization of product indexes). Yet another tasks may be asurvey asking participants to press a “yes” or “no” button or writeshort comments about participants' experiences or familiarity withcertain products and their satisfaction with the products. All thesetasks can be stored in a study content database 220, which can beretrieved by the virtual moderator module running on virtual moderatorserver 230 to forward to participants 120. Research module running onresearch server 210 can also be accessed by a client (e.g., a sponsor ofthe usability test) 171 who, like user experience researchers 181, candesign her own questionnaires since the client has a personal interestto the target web site under study. Client 171 can work together withuser experience researchers 181 to create tasks for usability testing.In an embodiment, client 171 can modify tasks or lists of questionsstored in the study content database 220. In another embodiment, client171 can add or delete tasks or questionnaires in the study contentdatabase 220. In yet another embodiment, client 171 may be userexperience researcher 181.

In some embodiment, one of the tasks may be open or closed card sortingstudies for optimizing the architecture and layout of the target website. Card sorting is a technique that shows how online users organizecontent in their own mind. In an open card sort, participants createtheir own names for the categories. In a closed card sort, participantsare provided with a predetermined set of category names. Client 171and/or user experience researcher 181 can create proprietary online cardsorting tool that executes card sorting exercises over large groups ofparticipants in a rapid and cost-effective manner. In an embodiment, thecard sorting exercises may include up to 100 items to sort and up to 12categories to group. One of the tasks may include categorizationcriteria such as asking participants questions “why do you group theseitems like this?.” Research module on research server 210 may combinecard sorting exercises and online questionnaire tools for detailedtaxonomy analysis. In an embodiment, the card sorting studies arecompatible with SPSS applications.

In an embodiment, the card sorting studies can be assigned randomly toparticipant 120. User experience (UX) researcher 181 and/or client 171may decide how many of those card sorting studies each participant isrequired to complete. For example, user experience researcher 181 maycreate a card sorting study within 12 tasks, group them in 4 groups of 3tasks and manage that each participant just has to complete one task ofeach group.

After presenting the thus created tasks to participants 120 throughvirtual moderator module (running on virtual moderator serer 230) andcommunication link 135, the actions/responses of participants will becollected in a data collecting module running on a data collectingserver 260 via a communication link 135′. In an embodiment,communication link 135′ may be a distributed computer network and sharethe same physical connection as communication link 135. This is, forexample, the case where data collecting module 260 locates physicallyclose to virtual moderator module 230, or if they share the usabilitytesting system's processing hardware. In the following description,software modules running on associated hardware platforms will have thesame reference numerals as their associated hardware platform. Forexample, virtual moderator module will be assigned the same referencenumeral as the virtual moderator server 230, and likewise datacollecting module will have the same reference numeral as the datacollecting server 260.

Data collecting module 260 may include a sample quality control modulethat screens and validates the received responses, and eliminatesparticipants who provide incorrect responses, or do not belong to apredetermined profile, or do not qualify for the study. Data collectingmodule 260 may include a “binning” module that is configured to classifythe validated responses and stores them into corresponding categories ina behavioral database 270.

Merely as an example, responses may include gathered web siteinteraction events such as clicks, keywords, URLs, scrolls, time ontask, navigation to other web pages, and the like. In one embodiment,virtual moderator server 230 has access to behavioral database 270 anduses the content of the behavioral database to interactively interfacewith participants 120. Based on data stored in the behavioral database,virtual moderator server 230 may direct participants to other pages ofthe target web site and further collect their interaction inputs inorder to improve the quantity and quality of the collected data and alsoencourage participants' engagement. In one embodiment, virtual moderatorserver may eliminate one or more participants based on data collected inthe behavioral database. This is the case if the one or moreparticipants provide inputs that fail to meet a predetermined profile.

Usability testing system 150 further includes an analytics module 280that is configured to provide analytics and reporting to queries comingfrom client 171 or user experience (UX) researcher 181. In anembodiment, analytics module 280 is running on a dedicated analyticsserver that offloads data processing tasks from traditional servers.Analytics server 280 is purpose-built for analytics and reporting andcan run queries from client 171 and/or user experience researcher 181much faster (e.g., 100 times faster) than conventional server system,regardless of the number of clients making queries or the complexity ofqueries. The purpose-built analytics server 280 is designed for rapidquery processing and ad hoc analytics and can deliver higher performanceat lower cost, and, thus provides a competitive advantage in the fieldof usability testing and reporting and allows a company such as UserZoom(or Xperience Consulting, SL) to get a jump start on its competitors.

In an embodiment, research module 210, virtual moderator module 230,data collecting module 260, and analytics server 280 are operated inrespective dedicated servers to provide higher performance. Client(sponsor) 171 and/or user experience research 181 may receive usabilitytest reports by accessing analytics server 280 via respective links 175′and/or 185′. Analytics server 280 may communicate with behavioraldatabase via a two-way communication link 272.

In an embodiment, study content database 220 may include a hard diskstorage or a disk array that is accessed via iSCSI or Fibre Channel overa storage area network. In an embodiment, the study content is providedto analytics server 280 via a link 222 so that analytics server 280 canretrieve the study content such as task descriptions, question texts,related answer texts, products by category, and the like, and generatetogether with the content of the behavioral database 270 comprehensivereports to client 171 and/or user experience researcher 181.

Shown in FIG. 2 is a connection 232 between virtual moderator server 230and behavioral database 270. Behavioral database 270 can be a networkattached storage server or a storage area network disk array thatincludes a two-way communication via link 232 with virtual moderatorserver 230. Behavioral database 270 is operative to support virtualmoderator server 230 during the usability testing session. For example,some questions or tasks are interactively presented to the participantsbased on data collected. It would be advantageous to the user experienceresearcher to set up specific questions that enhance the usabilitytesting if participants behave a certain way. If a participant decidesto go to a certain web page during the study, the virtual moderatorserver 230 will pop up corresponding questions related to that page; andanswers related to that page will be received and screened by datacollecting server 260 and categorized in behavioral database server 270.In some embodiments, virtual moderator server 230 operates together withdata stored in the behavioral database to proceed the next steps.Virtual moderator server, for example, may need to know whether aparticipant has successfully completed a task, or based on the datagathered in behavioral database 270, present another tasks to theparticipant.

Referring still to FIG. 2, client 171 and user experience researcher 181may provide one or more sets of questions associated with a target website to research server 210 via respective communication link 175 and185. Research server 210 stores the provided sets of questions in astudy content database 220 that may include a mass storage device, ahard disk storage or a disk array being in communication with researchserver 210 through a two-way interconnection link 212. The study contentdatabase may interface with virtual moderator server 230 through acommunication link 234 and provides one or more sets of questions toparticipants via virtual moderator server 230.

FIG. 3A is a flow diagram of an exemplary process of interfacing withpotential candidates and prescreening participants for the usabilitytesting according to one embodiment of the present invention. Theprocess starts at step 310. Initially, potential candidates for theusability testing may be recruited by email, advertisement banners,pop-ups, text layers, overlays, and the like (step 312). The number ofcandidates who have accepted the invitation to the usability test willbe determined at step 314. If the number of candidates reaches apredetermined target number, then other candidates who have signed uplate may be prompted with a message thanking for their interest and thatthey may be considered for a future survey (shown as “quota full” instep 316). At step 318, the usability testing system further determineswhether the participants' browser comply with a target web site browser.For example, user experience researchers or the client may want to studyand measure a web site's usability with regard to a specific web browser(e.g., Microsoft Edge) and reject all other browsers. Or in other cases,only the usability data of a web site related to Opera or Chrome will becollected, and Microsoft Edge or FireFox will be rejected at step 320.At step 322, participants will be prompted with a welcome message andinstructions are presented to participants that, for example, explainhow the usability testing will be performed, the rules to be followed,and the expected duration of the test, and the like. At step 324, one ormore sets of screening questions may be presented to collect profileinformation of the participants. Questions may relate to participants'experience with certain products, their awareness with certain brandnames, their gender, age, education level, income, online buying habits,and the like. At step 326, the system further eliminates participantsbased on the collected information data. For example, only participantswho have used the products under study will be accepted or screened out(step 328). At step 330, a quota for participants having a targetprofile will be determined. For example, half of the participants mustbe female, and they must have online purchase experience or havepurchased products online in recent years.

FIG. 3B is a flow diagram of an exemplary process for gatheringusability data of a target web site according to an embodiment of thepresent invention. At step 334, the target web site under test will beverified whether it includes a proprietary tracking code. In anembodiment, the tracking code is a UserZoom JavaScript code that pop-upsa series of tasks to the pre-screened participants. If the web siteunder study includes a proprietary tracking code (this corresponds tothe scenario shown in FIG. 1C), then the process proceeds to step 338.Otherwise, a virtual tracking code will be inserted to participants'browser at step 336. This corresponds to the scenario described above inFIG. 1A.

The following process flow is best understood together with FIG. 2. Atstep 338, a task is described to participants. The task can be, forexample, to ask participants to locate a color printer below a givenprice. At step 340, the task may redirect participants to a specific website such as eBay, HP, or Amazon.com. The progress of each participantin performing the task is monitored by a virtual study moderator at step342. At step 344, responses associated with the task are collected andverified against the task quality control rules. The step 344 may beperformed by the data collecting module 260 described above and shown inFIG. 2. Data collecting module 260 ensures the quality of the receivedresponses before storing them in a behavioral database 270 (FIG. 2).Behavioral database 270 may include data that the client and/or userexperience researcher want to determine such as how many web pages aparticipant viewed before selecting a product, how long it took theparticipant to select the product and complete the purchase, how manymouse clicks and text entries were required to complete the purchase andthe like. A number of participants may be screened out (step 346) duringstep 344 for non-complying with the task quality control rules and/orthe number of participants may be required to go over a series oftraining provided by the virtual moderator module 230. At step 348,virtual moderator module 230 determines whether or not participants havecompleted all tasks successfully. If all tasks are completedsuccessfully (e.g., participants were able to find a web page thatcontains the color printer under the given price), virtual moderatormodule 230 will prompt a success questionnaire to participants at step352. If not, then virtual moderator module 230 will prompt an abandon orerror questionnaire to participants who did not complete all taskssuccessfully to find out the causes that lead to the incompletion.Whether participants have completed all task successfully or not, theywill be prompted a final questionnaire at step 356.

FIG. 3C is a flow diagram of an exemplary process for card sortingstudies according to one embodiment of the present invention. At step360, participants may be prompted with additional tasks such as cardsorting exercises. Card sorting is a powerful technique for assessinghow participants or visitors of a target web site group related conceptstogether based on the degree of similarity or a number of sharedcharacteristics. Card sorting exercises may be time consuming. In anembodiment, participants will not be prompted all tasks but only arandom number of tasks for the card sorting exercise. For example, acard sorting study is created within 12 tasks that is grouped in 6groups of 2 tasks. Each participant just needs to complete one task ofeach group. It should be appreciated to one person of skill in the artthat many variations, modifications, and alternatives are possible torandomize the card sorting exercise to save time and cost. Once the cardsorting exercises are completed, participants are prompted with aquestionnaire for feedback at step 362. The feedback questionnaire mayinclude one or more survey questions such as a subjective rating oftarget web site attractiveness, how easy the product can be used,features that participants like or dislike, whether participants wouldrecommend the products to others, and the like. At step 364, the resultsof the card sorting exercises will be analyzed against a set of qualitycontrol rules, and the qualified results will be stored in thebehavioral database 270. In an embodiment, the analyze of the result ofthe card sorting exercise is performed by a dedicated analytics server280 that provides much higher performance than general-purpose serversto provide higher satisfaction to clients. If participants complete alltasks successfully, then the process proceeds to step 368, where allparticipants will be thanked for their time and/or any reward may bepaid out. Else, if participants do not comply or cannot complete thetasks successfully, the process proceeds to step 366 that eliminates thenon-compliant participants.

FIG. 4 illustrates an example of a suitable data processing unit 400configured to connect to a target web site, display web pages, gatherparticipant's responses related to the displayed web pages, interfacewith a usability testing system, and perform other tasks according to anembodiment of the present invention. System 400 is shown as including atleast one processor 402, which communicates with a number of peripheraldevices via a bus subsystem 404. These peripheral devices may include astorage subsystem 406, including, in part, a memory subsystem 408 and afile storage subsystem 410, user interface input devices 412, userinterface output devices 414, and a network interface subsystem 416 thatmay include a wireless communication port. The input and output devicesallow user interaction with data processing system 402. Bus system 404may be any of a variety of bus architectures such as ISA bus, VESA bus,PCI bus and others. Bus subsystem 404 provides a mechanism for enablingthe various components and subsystems of the processing device tocommunicate with each other. Although bus subsystem 404 is shownschematically as a single bus, alternative embodiments of the bussubsystem may utilize multiple busses.

User interface input devices 412 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a barcode scanner, a touch screen incorporated into thedisplay, audio input devices such as voice recognition systems,microphones, and other types of input devices. In general, use of theterm input device is intended to include all possible types of devicesand ways to input information to processing device. User interfaceoutput devices 414 may include a display subsystem, a printer, a faxmachine, or non-visual displays such as audio output devices. Thedisplay subsystem may be a cathode ray tube (CRT), a flat-panel devicesuch as a liquid crystal display (LCD), or a projection device. Ingeneral, use of the term output device is intended to include allpossible types of devices and ways to output information from theprocessing device.

Storage subsystem 406 may be configured to store the basic programmingand data constructs that provide the functionality in accordance withembodiments of the present invention. For example, according to oneembodiment of the present invention, software modules implementing thefunctionality of the present invention may be stored in storagesubsystem 406. These software modules may be executed by processor(s)402. Such software modules can include codes configured to access atarget web site, codes configured to modify a downloaded copy of thetarget web site by inserting a tracking code, codes configured todisplay a list of predefined tasks to a participant, codes configured togather participant's responses, and codes configured to causeparticipant to participate in card sorting exercises. Storage subsystem406 may also include codes configured to transmit participant'sresponses to a usability testing system.

Memory subsystem 408 may include a number of memories including a mainrandom access memory (RAM) 418 for storage of instructions and dataduring program execution and a read only memory (ROM) 420 in which fixedinstructions are stored. File storage subsystem 410 provides persistent(non-volatile) storage for program and data files, and may include ahard disk drive, a floppy disk drive along with associated removablemedia, a Compact Disk Read Only Memory (CD-ROM) drive, an optical drive,removable media cartridges, and other like storage media.

Now that systems and methods of usability testing have been described ata high level, attention will be directed to a particular set ofembodiments of the systems and methods for user experience testing thatallows for advanced insight generation. This begins with a usabilitytesting system 150 as seen in relation to FIG. 5. In this substantiationof the usability testing system 150 a number of subcomponents are seenas logically connected with one another, including an interface 510 foraccessing the results 570 which may be stored internally or in anexternal data repository. The interface is also configured to couplewith the network 560, which most typically is the Internet, aspreviously discussed.

The other significant components of the user experience testing system150 includes a study generation module 520, a recruitment engine 530, astudy administrator 540 and a research module 550, each of which will bedescribed in greater detail below. Each of the components of the userexperience testing systems 150 may be physically or logically coupled,allowing for the output of any given component to be used by the othercomponents as needed.

Turning to FIG. 6, the study generation module 520 is provided ingreater detail. An offline template module 521 provides a system userwith templates in a variety of languages (pre-translated templates) forstudy generation, screener questions and the like, based upon studytype. Users are able to save any screener question, study task, etc. forusage again at a later time or in another study.

In some embodiments a user may be able to concurrently design anunlimited number of studies, but is limited in the deployment of thestudies due to the resource expenditure of participants andcomputational expense of the study insight generation. As such, asubscription administrator 523 manages the login credentialing, studyaccess and deployment of the created studies for the user. In someembodiments, the user is able to have subscriptions that scale inpricing based upon the types of participants involved in a stud, and thenumber of studies concurrently deployable by the user/client.

The translation engine 525 may include machine translation services forstudy templates and even allow on the fly question translations. Ascreener module 527 is configured to allow for the generation ofscreener questions to weed through the participants to only those thatare suited for the given study. This may include basic Booleanexpressions with logical conditions to select a particular demographicfor the study. However, the screener module 527 may also allow foradvanced screener capabilities where screener groups and quotas aredefined, allowing for advanced logical conditions to segmentparticipants. For example, the study may wish to include a group of 20women between the ages of 25-45 and a group of men who are between theages of 40-50 as this may more accurately reflect the actual purchasingdemographic for a particular retailer. A single participant screeningwould be unable to generate this mix of participants, so the advancedscreener interface is utilized to ensure the participants selected meetthe user's needs for the particular study.

Turning now to FIG. 7, a more detailed illustration of the recruitmentengine 530 is provided. The recruitment engine 530 is responsible forthe recruiting and management of participants for the studies.Generally, participants are one of three different classes: 1) corepanel participants, 2) general panel participants, and 3) clientprovided participants. The core panel participants are compensated at agreater rate, but must first be vetted for their ability and willingnessto provide comprehensive user experience reviews. Significantdemographic and personal information can be collected for these corepanel participants, which can enable powerful downstream analytics. Thecore panel vetting engine 531 collects public information automaticallyfor the participants as well as eliciting information from theparticipant to determine if the individual is a reliable panelists.Traits like honesty and responsiveness may be ascertained by comparingthe information derived from public sources to the participant suppliedinformation. Additionally, the participant may provide a video sample ofa study. This sample is reviewed for clarity and communicationproficiency as part of the vetting process. If a participant issuccessfully vetted they are then added to a database of available corepanelists. Core panelists have an expectation of reduced privacy, andmay pre-commit to certain volumes and/or activities.

Beyond the core panel is a significantly larger pool of participants ina general panel participant pool. This pool of participants may haveactivities that they are unwilling to engage in (e.g., audio and videorecording for example), and are required to provide less demographic andpersonal information than core panelists. In turn, the general panelparticipants are generally provided a lower compensation for their timethan the core panelists. Additionally, the general panel participantsmay be a shared pooling of participants across many user experience andsurvey platforms. This enables a demographically rich and large pool ofindividuals to source from. A large panel network 533 manages thisgeneral panel participant pool.

Lastly, the user or client may already have a set of participants theywish to use in their testing. For example, if the user experience for anemployee benefits portal is being tested, the client will wish to testthe study on their own employees rather than the general public.

A reimbursement engine 535 is involved with compensating participantsfor their time (often on a per study basis). Different studies may be‘worth’ differing amounts based upon the requirements (e.g., videorecording, surveys, tasks, etc.) or the expected length to completion.Additionally, the compensation between general panelists and corepanelists may differ even for the same study. Generally, client suppliedparticipants are not compensated by the reimbursement engine 535 as thecompensation (if any) is directly negotiated between the client and theparticipants.

Turning now to FIG. 8, a more detailed view of the study administrator540 is provided. Unlike many other user experience testing programs, thepresently disclosed systems and methods include the ability to recordparticular activities by the user. A recording enabler 541 allows forthe collection of click-flow information, audio collection and evenvideo recording. In the event of audio and/or video recording therecording only occurs during the study in order to preserve participantprivacy, and to focus attention on only time periods that will provideinsights into the user experience. Thus, while the participant isengaged in screening questions or other activities recording may bedisabled to prevent needless data accumulation. Recording only occursafter user acceptance (to prevent running afoul of privacy laws andregulations), and during recording the user may be presented with aclear indication that the session is being recorded. For example theuser may be provided a thumbnail image of the video capture, in someembodiments. This provides notice to the user of the video recording,and also indicates video quality and field of view information, therebyallowing them to readjust the camera if needed or take other necessaryactions (avoiding harsh backlight, increasing ambient lighting, etc.).

The screening engine 543 administers the generated screener questionsfor the study. Screener questions, as previously disclosed, includesquestions to the potential participants that may qualify or disqualifythem from a particular study. For example, in a given study, the usermay wish to target men between the ages of 21 and 35, for example.Questions regarding age and gender may be used in the screener questionsto enable selection of the appropriate participants for the given study.Additionally, based upon the desired participant pool being used, theparticipants may be pre-screened by the system based upon knowndemographic data. For the vetted core panelists the amount of personaldata known may be significant, thereby focusing in on eligibleparticipants with little to no additional screener questions required.For the general panel population, however, less data is known, and oftenall but the most rudimentary qualifications may be performedautomatically. After this qualification filtering of the participants,they may be subjected to the screener questions as discussed above.

In some embodiments it may be desirable to interrupt a study in progressin order to interject a new concept, offer or instruction. Particularly,in a mobile application there can be a software developer kit (SDK) thatenables the integration into the study and interruption of the userin-process. The study interceptor 545 manages this interruptiveactivity. Interruption of the user experience allows for immediatefeedback testing or prompts to have the participant do some otheractivity. For example, the interrupt may be configured to trigger whensome event or action is taken, such as the participant visiting aparticular URL or meeting a determined threshold (e.g. having two itemsin their shopping cart). The interruption allows the participant to beeither redirected to another parallel user experience, or be prompted toagree to engage in a study or asked to answer a survey or the like.

Lastly, the study may include one or more events to occur in order tovalidate its successful completion. A task validator 547 tracks thesemetrics for study completion. Generally, task validation falls intothree categories: 1) completion of a particular action (such as arrivingat a particular URL, URL containing a particular keyword, or the like),2) completing a task within a time threshold (such as finding a productthat meets criteria within a particular time limit), and 3) by question.Questions may include any definition of success the study designer deemsrelevant. This may include a simple “were you successful in the task?”style question, or a more complex satisfaction question with multiplegradient answers, for example.

Turning now to FIG. 9, the research module 550 is provided in greaterdetail. Compared to traditional user experience study platforms, thepresent systems and methods particularly excel at providing timely andaccurate insights into a user's experience, due to these research tools.The research module includes basic functionalities, such as playback ofany video or audio recordings by the playback module 551. This module,however, may also include a machine transcription of the audio, which isthen time synchronized to the audio and/or video file. This allows auser to review and search the transcript (using keywords or the like)and immediately be taken to the relevant timing within the recording.And of the results may be annotated using an annotator 559 as well. Thisallows, for example the user to select a portion of the writtentranscription and provide an annotation relevant to the study results.The system then automatically can use the timing data to generate anedited video/audio clip associated with the annotation. If the userlater searches the study results for the annotation, this auto-generatedclip may be displayed for viewing.

In addition to the video and/or audio recordings, the clickstream forthe participant is recorded and mapped out as a branched tree, by theclick stream analyzer 553. This may be aggregated with otherparticipants' results for the study, to provide the user an indicationof what any specific participant does to complete the assigned task, orsome aggregated group generally does. The results aggregator 555likewise combines task validation findings into aggregate numbers foranalysis.

All results may be searched and filtered by a filtering engine 557 basedupon any delineator. For example, a user may desire to know what thepain points of a given task are, and thus filters the results only byparticipants that failed to complete the task. Trends in the clickstreamfor these individuals may illustrate common activities that result infailure to complete the task. For example, if the task is to find alaptop computer with a dedicated graphics card for under a set price,and the majority of people who fail to successfully complete this taskend up stuck in computer components due to typing in a search for“graphics card” this may indicate that the search algorithm requiresreworking to provide a wider set of categories of products, for example.

As noted above, the filtering may be by any known dimension (not simplysuccess or failure events of a task). For example, during screening oras part of a survey attending the study, income levels, gender,education, age, shopping preferences, etc. may all be discovered. It isalso possible that the participant pool includes some of thisinformation in metadata associated with the participant as well. Any ofthis information may be used to drill down into the results filtering.For example it may be desired to filter for only participants over acertain age. If after a certain age success rates are found to drop offsignificantly, for example, it may be that the font sizing is too small,resulting in increased difficulty for people with deterioratingeyesight.

Likewise, any of the results may be subject to annotations. Annotationsallow for different user reviewers to collectively aggregate insightsthat they develop by reviewing the results, and allows for filtering andsearching for common events in the results.

All of the results activities are additionally ripe for machine learninganalysis using deep learning. For example, the known demographicinformation may be fed into a recursive neural network (RNN) orconvoluted neural network (CNN) to identify which features arepredictive of a task being completed or not. Even more powerful is theability for the clickstream to be fed as a feature set into the neuralnetwork to identify trends in click flow activity that are problematicor result in a decreased user experience.

Turning now to FIG. 10, a flow diagram of the process of user experiencestudy testing is provided generally at 1000. At a high level thisprocess includes three basic stages: the generation of the study (at1010) the administration of the study (at 1020) and the generation ofthe study insights (at 1030). Earlier FIGS. 3A-C touched upon the studyadministration, and is intended to be considered one embodiment thereof.

FIG. 11 provides a more detailed flow diagram of the study generation1010. As noted before, the present systems and methods allows forimproved study generation by the usage of study templates which areselected (at 1110) based upon the device the study is to be implementedon, and the type of study that is being performed. Study templates maycome in alternate languages as well, in some embodiments. Study typesgenerally include basic usability testing, surveys, card sort, treetest, click test, live intercept and advanced user insight research. Thebasic usability test includes audio and/or video recordings for arelatively small number of participants with feedback. A survey, on theother hand, leverages large participant numbers with branched surveyquestions. Surveys may also include randomization and double blindstudies. Card sort, as discussed in great detail previously, includesopen or closed card sorting studies. Tree tests assess the ease in whichan item is found in a website menu by measuring where users expect tolocate specific information. This includes uploading a tree menu andpresenting the participant with a task to find some item within themenu. The time taken to find the item, and rate of successful versusunsuccessful queries into different areas of the tree menu are collectedas results.

Click test measures first impressions and defines success areas on astatic image as a heat map graph. In the click test the participant ispresented with a static image (this may include a mock layout of awebsite/screenshot of the webpage, an advertising image, an array ofimages or any other static image) and is presented a text prompt. Thetext prompt may include questions such “Which image makes you thehungriest?” or “select the tab where you think deals on televisions arefound.” The location and time the user clicks on the static image isrecorded for the generation of a heat map. Clicks that take longer(indicating a degree of uncertainty on behalf of the participant) areweighted as less strong, whereas immediate selection indicates that thereaction by the participant is surer. Over time the selections ofvarious participants may be collected. Where many participants select ananswer to a particular prompt in the same place relatively rapidly thereis a darker heat map indicator. Where participants select variouslocations, the heat map will show a more diffuse result. Consistentlocation, but longer delay in the selection will also result in aconcentration on the heat map, but of a lighter color, indicating thedegree of insecurity by the participants.

Additionally, the user may be able to define regions on the static imagethat are considered ‘answers’ to the prompted question. This may allowfor larger scale collection of success versus failure metrics, as wellas enabling follow-up activities, such as a survey or additional clicktest, based upon where the participant clicks on the image.

Lastly, advanced research includes a combination of the othermethodologies with logical conditions and task validation, and is thesubject of much of the below discussions. Each of these study typesincludes separate saved template designs.

Device type is selected next (at 1120). As noted before, mobileapplications enable SDK integration for user experience interruption,when this study type is desired. Additionally, the device type isimportant for determining recording ability/camera capability (e.g., amobile device will have a forward and reverse camera, whereas a laptopis likely to only have a single recording camera, whereas a desktop isnot guaranteed to have any recording device) and the display type thatis particularly well suited for the given device due to screen sizeconstraints and the like.

The study tracking and recording requirements are likewise set (at1130). Further, the participant types are selected (at 1140). Theselection of participants may include a selection by the user to usetheir own participants, or rely upon the study system for providingqualifies participants. If the study system is providing theparticipants, a set of screener questions are generated (at 1150). Thesescreener questions may be saved for later usage as a screener profile.The core participants and larger general panel participants may bescreened until the study quota is filled.

Next the study requirements are set (at 1160). Study requirements maydiffer based upon the study type that was previously selected. Forexample, the study questions are set for a survey style study, oradvanced research study. In basic usability studies and research studiesthe task may likewise be defined for the participants. For tree teststhe information being sought is defined and the menu uploaded. For clicktest the static image is selected for usage. Lastly, the successvalidation is set (at 1170) for the advanced research study.

After study generation, the study may be implemented, as shown ingreater detail at 1020 of FIG. 12. Study implementation begins withscreening of the participants (at 1210). This includes initiallyfiltering all possible participants by known demographic or personalinformation to determine potentially eligible individuals. For example,basic demographic data such as age range, household income and gendermay be known for all participants. Additional demographic data such aseducation level, political affiliation, geography, race, languagesspoken, social network connections, etc. may be compiled over time andincorporated into embodiments, when desired. The screener profile mayprovide basic threshold requirements for these known demographics,allowing the system to immediately remove ineligible participants fromthe study. The remaining participants may be provided access to thestudy, or preferentially invited to the study, based upon participantworkload, past performance, and study quota numbers. For example, alimited number (less than 30 participants) video recorded study thattakes a long time (greater than 20 minutes) may be provided out on aninvitation basis to only core panel participants with proven historiesof engaging in these kinds of studies. In contrast, a large surveyrequiring a thousand participants that is expected to only take a fewminutes may be offered to all eligible participants.

The initially eligible participants are then presented with the screenerquestions. This two-phased approach to participant screening ensuresthat participants are not presented with studies they would never beeligible for based upon their basic demographic data (reducingparticipant fatigue and frustration), but still enables the user toconfigure the studies to target a particular participant based upon veryspecific criteria (e.g., purchasing baby products in the past week forexample).

After participants have been screened and are determined to still meetthe study requirements, they are asked to accept the study terms andconditions (at 1220). As noted before, privacy regulations play an everincreasing role in online activity, particularly if the individual isbeing video recorded. Consent to such recordings is necessitated bythese regulations, as well as being generally a best practice.

After conditions of the study are accepted, the participant may bepresented with the study task (at 1230) which, again, depends directlyupon the study type. This may include navigating a menu, finding aspecific item, locating a URL, answering survey questions, providing anaudio feedback, card sorting, clicking on a static image, or somecombination thereof. Depending upon the tasks involved, the clickstreamand optionally audio and/or video information may be recorded (at 1240).The task completion is likewise validated (at 1250) if the successcriteria is met for the study. This may include task completion in aparticular time, locating a specific URL, answering a question, or acombination thereof.

After study administration across the participant quota, insights aregenerated for the study based upon the results, as seen at 1030 of FIG.13. Initially the study results are aggregated (at 1310). This includesgraphing the number of studies that were successful, unsuccessful andthose that were abandoned prior to completion. Confidence intervals maybe calculated for these graphs. Similarly, survey question results maybe aggregated and graphed. Clickstream data may be aggregated and thelikelihood of any particular path may be presented in a branchedgraphical structure. Aggregation may include the totality of allresults, and may be delineated by any dimension of the study.

When an audio or video recording has been collected for the study, theserecordings may be transcribed using machine voice to text technology (at1320). Transcription enables searching of the audio recordings bykeywords. The transcriptions may be synchronized to the timing of therecording, thus when a portion of the transcription is searched, therecording will be set to the corresponding frames. This allows for easyreview of the recording, and allows for automatic clip generation byselecting portions of the transcription to highlight and tag/annotate(at 1330). The corresponding video or audio clip is automatically editedthat corresponds to this tag for easy retrieval. The clip can likewisebe shared by a public URL for wider dissemination. Any portion of theresults, such as survey results and clickstream graphs, may similarly beannotated for simplified review.

As noted, clickstream data is analyzed (at 1340). This may include therendering of the clickstream graphical interface showing what variousparticipants did at each stage of their task. As noted before, deeplearning neural networks may consume these graphs to identify ‘points ofconfusion’ which are transition points that are predictive of a failedoutcome.

All the results are filterable (at 1350) allowing for complex analysisacross any study dimension. Here too, machine learning analysis may beemployed, with every dimension of the study being a feature, to identifywhat elements (or combination thereof) are predictive of a particularoutcome. This information may be employed to improve the design ofsubsequent website designs, menus, search results, and the like.

Although not illustrated, video recording also enables additionalanalysis not previously available, such as eye movement tracking andimage analysis techniques. For example, a number of facial recognitiontools are available for emotion detection. Key emotions such as anger,frustration, excitement and contentment may be particularly helpful indetermining the user's experience. A user who exhibits frustration witha task, yet still completes the study task may warrant review despitethe successful completion. Results of these advanced machine learningtechniques may be automatically annotated into the recording for searchby a user during analysis.

While the above discussion has been focused upon testing the userexperience in a website for data generation, it is also possible thatthese systems and methods are proactively deployed defensively againstcompetitors who are themselves engaging in user experience analysis.This includes first identifying when a user experience test is beingperformed, and taking some reaction accordingly. Red-flag behaviors,such as redirection to the client's webpage from a competitive userexperience analytics firm is one clear behavior. Others could include apattern of unusual activity, such as a sudden increase in a verydiscrete activity for a short duration.

Once it is determined that a client's website has been targeted for somesort of user experience test, the event is logged. At a minimum thissort of information if helpful to the client in planning their own userexperience tests, and understanding what their competitors are doing.However, in more extreme situations, alternate web portals may beemployed to obfuscate the analysis being performed.

Turning now to FIGS. 14-24, 25A-25F, 26, 27A-27D, 28A-28B and 29,example screenshots of the operation of the user experience study systemare provided. Particularly at FIG. 14, an initial study generation siteis provided at 1400 which allows the user to select the type of studythey wish to generate. The study types have been previously discussed insignificant detail, and allow for template selection based upon the userinput.

Upon selection of the study type, the user is presented with a screen todetermine what device type the study will occur on, at 1500 of FIG. 15.The device type ensures that the study interface is properly adapted forthe screen size and device capabilities. Additionally, if a mobileapplication is selected, the use of SDK integration enables userexperience interruptions.

After the device type has been selected, the project/study details arerequested from the user, at screen 1600 of FIG. 16. This includesassigning an internal and external name for the study, applying relevantlabels, inputting notes and other information regarding the goals of thestudy, and selecting the recording level desired and the participantlanguage. A requirement for consent is also available based upon thejurisdiction the study is deployed in and/or the level of datacollected.

After the study is thus created, the participant selection screen 1700is presented to the user, as seen in FIG. 17. Generally the options arefor the user to supply their own participant pool, or utilize the poolof participants available to the user experience system. Again, thispool of participants includes the core panel as well as the generalpanel of participants.

On the left side of the screen it can be seen that a screen navigationlink is available. If selected, the user is redirected to a screenerinterface, seen at 1800 of FIG. 18. On the screener interface the usermay select questions used to screen potential participants. Thequestions available include single answer (radial button) answers, dropdown single answer questions, multiple answer questions (check box),rating scaled questions, and text answer questions. An example of thegeneration of a one answer question is provided in relation to theinterface 1900 of FIG. 19. The user inputs the question text andpossible answers. Randomization events may be selected for the answersto avoid errors associated with answer patterns. The logical result ofthe answer selection allows for early termination, or branched questionsequences.

After the question(s) are generated, they may be linked together inlogical conditions in order to produce a screener group, as seen ininterface 2000 of FIG. 20. This allows for the questions to be arrangedin “or” and “and” clusters.

When the system is providing the participants, the user is stillrequested to provide information regarding segments that they wish to beincluded in the study, as seen in FIG. 21 at interface 2100. The segmentis provided a name and a number of desired participants. Additionally,basic data such as location, gender, age and household income may be setto ensure that only eligible participants are presented with the study(or invited to join the study). This interface 2100 is simpler than themore invasive screener questions, and is generally employed in thealternative, when a faster and less involved participant list is needed(multiple participant groups are not generated in this embodiment).

After the participants are determined, the user configures the actualtasks or questions involved in the study, as seen in the interface 2200of FIG. 22. The tasks may include navigation tasks (either created fromscratch or imported from this or a previous study), as seen, or mayinclude any of the other tasks noted previously by pulling down on thedrop-down menu. Other tasks may include click tests, card sorts,surveys, etc. This interface additionally allows the user to indicatethe order of the tasks.

After the task is created, it is configured via a task configurationinterface 2300 of FIG. 23. For a navigation task as seen, this mayinclude putting the participant at an initialization URL, or merelybeginning from a webpage the participant was at in the previous step.The task is titled and described. For example, the task may state “findand update your contact information.” A taskbar description is alsodescribed, and can include a success and abandonment selection option.In alternate embodiments, the taskbar may instead only includeabandonment as an option, and success is defined by successfullycompleting the validation requirements. At the top of the interface, theuser has access to a validation tab for configuring validationrequirements, and a recording tab, where video recording options areset.

After all the tasks are set in this manner the study may be launched andmade available to participants for testing. The results of the testingare compiled in a monitoring interface as seen at 2400 of FIG. 24. Thisinterface shown the number of participants initially involved. In thiscase an original number of 24 participants were invited for the study.Of the 24, seven were excluded from the study due to not meeting thescreener question requirements. An additional two individuals exited thestudy (abandoned it) within the welcome screen. The remaining 15participants completed the study.

Turning now to FIGS. 25A-25F, details of the study results are providedin more detail. For example, as seen at 2500A of FIG. 25A, it can beseen that all 15 of the participants came from a single participantsegment. Other information, such as location of the participants areseen, as well as age, gender and household income levels (at 2500B ofFIG. 25B). Results from the screener questions are also presented, asseen in 2500C of FIG. 25C. After screener question results, the initialquestionnaire results are presented, as seen at 2500D of FIG. 25D.

Next, results of task completion are presented. Here is can be seen at2500E of FIG. 25E that two tasks were presented to the participants, andthat 13 of the participants were able to successfully complete the firsttask, and 14 were able to complete the second task. Each task is thenanalyzed in greater detail, including effectiveness, and as seen at2500F of FIG. 25F, the number of page views, number of clicks required,and (not sown) timing for task completion are likewise presented. Insome cases, these results may be provided with confidence intervals thatpredict what 95% of individuals who are presented the task will fallwithin.

In this example, it is clear that the participants that were unable tosuccessfully complete this task were sidetracked and ended up lost inthe website, clicking on page after page without finding what they werelooking for. The click flow graph (not illustrated) may be leveraged bythe user to determine at what stage the participants that wereunsuccessful derailed from the successful pathway(s). The timing ofthese flows may then be used to review the video recordings for theparticipants to better understand what “went wrong” in their experience.FIG. 26 provides an example screenshot of a review pane 2600 for thevideo clips. The left side navigation menu illustrates how the videoscan be filtered by tasks, marks, effectiveness, camera options, audiooptions, ones that have been viewed (or not), video that have beenclipped and availability.

FIG. 27A provides an example screenshot 2700A of a first heat mapgenerated for a webpage screenshot image using a click test. In thisexample, the participants were given a prompt of “Where would you clickto get keychains?” In this first heat map only the click locations areillustrated. In this example, the user defined the “keychain” hyperlinkas a successful ‘answer’ to the prompt/question. The results from theclick test can thus be collected and illustrated in a simple metricsinterface as seen in image 2700B of FIG. 27B. Here is can be seen that12 of the 15 participants selected the correct link, whereas 3participants clicked at some other location.

FIGS. 27C and 27D provide additional heat map interfaces that capturenot only click locations, but also speed of the clicks, as seen atimages 2700C and 2700D, respectively. The screenshot of 2700D is ablackout heat map which accentuates the click locations for uservisibility.

Moving on, FIG. 28A provides a screenshot 2800A of a user uploading atree structure menu into the system for a tree test. The user is thenprompted, when shown only the top level of the tree, to find aparticular item in the tree structure. In one example tree test theparticipants are requested to find olive oil. The results of this clicktest may be seen in relation to FIG. 28B at screenshot 2800B. Here itcan be seen, both in terms of percentages and raw numbers ofparticipants, where people looked for the product ‘olive oil’. Themajority of participants rightfully went to the ‘ingredients’ categoryand then to ‘oils’. However, a sizable group of participants mistakenlyselected ‘condiments and sauces’ and then ‘olives’ instead. Thisprovides insights into potential locations for redirection, crossadvertising of products, or in the extreme updating the way the treestructure is laid out.

Lastly, FIG. 29 provides an example screenshot of a dendrogram 2900 froma card sorting or tree test activity. Dendrograms provide results in atree structure that allows for selection of a node within the decisionprocess to determine what portion of total participants falls below theselected node. In this example 77.7% of participants selected luggagestorage, express check in, slippers and private bathrooms during thecard sort involved in this example.

Some portions of the above detailed description may be presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is, here and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. It has proven convenient at times, principally for reasonsof common usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers, or the like.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the methods of some embodiments. The requiredstructure for a variety of these systems will appear from thedescription below. In addition, the techniques are not described withreference to any particular programming language, and variousembodiments may, thus, be implemented using a variety of programminglanguages.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a client-server network environment or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a virtualmachine, a personal computer (PC), a tablet PC, a laptop computer, aset-top box (STB), a personal digital assistant (PDA), a cellulartelephone, an iPhone, a Blackberry, a processor, a telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine.

While the machine-readable medium or machine-readable storage medium isshown in an exemplary embodiment to be a single medium, the term“machine-readable medium” and “machine-readable storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store the one or more sets of instructions. The term“machine-readable medium” and “machine-readable storage medium” shallalso be taken to include any medium that is capable of storing, encodingor carrying a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresently disclosed technique and innovation.

In general, the routines executed to implement the embodiments of thedisclosure may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer, and when read andexecuted by one or more processing units or processors in a computer,cause the computer to perform operations to execute elements involvingthe various aspects of the disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution

While this invention has been described in terms of several embodiments,there are alterations, modifications, permutations, and substituteequivalents, which fall within the scope of this invention. Althoughsub-section titles have been provided to aid in the description of theinvention, these titles are merely illustrative and are not intended tolimit the scope of the present invention. It should also be noted thatthere are many alternative ways of implementing the methods andapparatuses of the present invention. It is therefore intended that thefollowing appended claims be interpreted as including all suchalterations, modifications, permutations, and substitute equivalents asfall within the true spirit and scope of the present invention.

What is claimed is:
 1. A method for generating intents from a userexperience study comprising: applying at least one screener question toa plurality of participants; subjecting the screened participants to atleast one task; recording clickstream and success data for eachparticipant engaging in the at least one task, wherein the clickstreamdata is mapped as a branched tree structure; aggregating the success andclickstream data for all the screened participants into aggregatedresults, wherein the aggregated results include a likelihood of a givenpath taken in the branched tree structure; filtering the aggregatedclickstream data by a configured dimension; and identifying features ofthe clickstream data that correspond to failure of the at least one taskby feeding the aggregated clickstream data into a neural network.
 2. Themethod of claim 1, further comprising recording at least one validationcriteria.
 3. The method of claim 2, wherein the at least one validationcriteria defines the success.
 4. The method of claim 1, furthercomprising recording video data for each of the screened participants.5. The method of claim 4, further comprising transcribing audio data inthe video data.
 6. The method of claim 5, further comprisingsynchronizing the transcribed audio data to times in the video data. 7.The method of claim 6, further comprising searching the transcribedaudio data.
 8. The method of claim 7, further comprising playing thevideo data at a time corresponding to the searched transcribed audiodata.
 9. The method of claim 6, further comprising selecting a portionof the transcribed audio data and automatically generating a video clipedited from the video data corresponding to the timing of the selectedportion.
 10. The method of claim 1, wherein the task includes at leastone of a navigation task, a survey, a click test, a card sorting, and atree test.
 11. A system for generating intents from a user experiencestudy comprising: a screener for applying at least one screener questionto a plurality of participants; an administrator for subjecting thescreened participants to at least one task; a results module forrecording clickstream and success data for each participant engaging inthe at least one task, wherein the clickstream data is mapped as abranched tree structure, aggregating the success and clickstream datafor all the screened participants into aggregated results, wherein theaggregated results include a likelihood of a given path taken in thebranched tree structure, and filtering the aggregated clickstream databy a configured dimension; and a neural network for identifying featuresof the clickstream data that correspond to failure of the at least onetask by consuming the aggregate clickstream data.
 12. The system ofclaim 11, wherein the results module is further for recording at leastone validation criteria.
 13. The system of claim 12, wherein the atleast one validation criteria defines the success.
 14. The system ofclaim 11, wherein the results module is further for recording video datafor each of the screened participants.
 15. The system of claim 14,further comprising a transcriber for transcribing audio data in thevideo data.
 16. The system of claim 15, wherein the results module isfurther for synchronizing the transcribed audio data to times in thevideo data.
 17. The system of claim 16, wherein the results module isfurther for searching the transcribed audio data.
 18. The system ofclaim 17, wherein the results module is further for playing the videodata at a time corresponding to the searched transcribed audio data. 19.The system of claim 16, an automated clip generator for selecting aportion of the transcribed audio data and automatically generating avideo clip edited from the video data corresponding to the timing of theselected portion.
 20. The system of claim 11, wherein the task includesat least one of a navigation task, a survey, a click test, a cardsorting, and a tree test.